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Abstract 

It is proved that the periodic point submonoid of a free inverse monoid 
endomorphism is always finitely generated. Using Chomsky's hierarchy 
of languages, we prove that the fixed point submonoid of an endomor- 
phism of a free inverse monoid can be represented by a context-sensitive 
language but, in general, it cannot be represented by a context-free 
language. 

1 Introduction 

The dynamical study of the automorphisms of a free group and their space 
of ends is a well established subject in discrete Dynamical Systems. Hence 
it is a natural issue to explore these problems in the more general setting 
of semigroups. In [3], Cassaigne and the second author studied finiteness 
conditions for the infinite fixed points of (uniformly continuous) endomor- 
phisms of monoids defined by special confluent rewriting systems, extending 
results known for free monoids [H] . This line of research was pursued by the 
second author in subsequent papers [171 13 US]. Recently, in a joint paper 
with Sykiotis [H], we considered the case of graph groups, studying the pe- 
riodic and fixed points of endomorphisms. Inspired by this work, we studied 
in [T^ the case of endomorphisms of trace monoids and their extensions to 
real traces. 

In this paper we consider the case of endomorphisms of free inverse 
monoids. Inverse monoids are a natural generalization of groups. Indeed, 
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while groups are represented by bijective functions, inverse semigroups are 
represented by partial injective functions. Although these structures seem 
to be close, the behavior of their respective free objects can be radically dif- 
ferent. In this paper we provide further evidence to this viewpoint. In fact, 
we show that the inverse submonoid of the points fixed by an endomorphism 
of a free inverse monoid is not in general finitely generated as in the group 
case. However, we can still frame the fixed point submonoid in Chomsky's 
hierarchy, at the context-sensitive level. A better outcome is achieved for 
the periodic point submonoid (finitely generated, hence rational) and for 
some related submonoids called radicals (context-free). 

The paper is organized as follows. Section [2] is devoted to preliminar- 
ies. In Section [3] we show that the inverse submonoid of periodic points 
is finitely generated, while the inverse submonoid of fixed points is not in 
general context-free. This is achieved by proving a gap theorem for endomor- 
phisms whose induced endomorphisms in the free group has no nontrivial 
fixed points in the boundary. In Section we introduce a family of partic- 
ular inverse submonoids called radicals, useful in describing the fixed points 
submonoid. We show that the radical behaves better than the fixed points 
submonoids by proving that the A^th radical is context-free for suitable A^. 
Using this result we finally frame the fixed point submonoid in the context- 
sensitive class. Finally, we raise some open problems in the last section. 

2 Preliminaries 

The reader is assumed to have some familiarity with the basics of formal 
language theory. For references, see [HE]. 

2.1 Free groups 

Let A be a finite alphabet and let A = Au A~^ be the involutive alphabet 
where A~^ is the set of formal inverses of A. The operator ~^ : A — ?> A~^ : 
a 1-^ is extended to an involution on the free monoid A* through 

1"^ = 1, {a~^)-^ = a, (uv)-^ = y-^u-^ {a e A; u,v £ A*). 

Let ~ be the congruence on A* generated by the relation {(aa~^, 1) | a S A}. 
The quotient Fa = A* / is the free group on A, and a : A* ^ Fa denotes 
the canonical homomorphism. 

We denote by Ra the set of all reduced words on A*, i.e. 

Ra = A*\\J A*aa-^A* 



2 



For each n € u € Ra is the (unique) reduced word ^-equivalent to u. 
We write also ua = u. As usual, we often identify the elements of with 
their reduced representatives. For B Q A*, B denotes the set of reduced 
words of B. 

We describe now the boundary of F4. Given u,v (z F^, written in reduced 
form, we denote hy u Av the longest common prefix of u and v. Let 

( N _ / 1^ ""I u ^ V 
[ +00 otherwise 

and define d{u,v) = 2~^("''') (using the convention 2~°° = 0). Then d is 
know as the prefix metric on Fa- In fact, d is an ultrametric since 

d{u,w) < ma'x.{d{u,v),d{v,w)} 

holds for all u,v,w G Fa- 

The metric space {FA,d) is not complete, but its completion admits 
a simple description: we add to Fa all the (right) infinite reduced words 
ffliffl2fl3 ... on ^. These new elements are called the boundary of Fa and the 
completion (which is indeed compact) is denoted by Fa- The metric on Fa 
admits the same description as d and is also denoted by d- 

By standard topology results (see [H Section XIV.6]), any uniformly 
continuous endomorphism 6 of Fa admits a (unique) continuous extension 
9 ■- Fa Fa- Thus, if (n„)„ is a sequence on Fa converging in Fa, then 

( lim Un)0 = lim UnO 

n— >-oo n— >oo 

by continuity. The uniformly continuous endomorphisms of Fa turn out 
to be the monomorphisms. For details on the boundary and endomorphism 
extensions on a general context (containing free groups as a particular case) , 
the reader is referred to [2]. 

2.2 Free inverse monoids 

A monoid M is said to be inverse if 

Vii € M 3lu~^ e M : {uu'^u = u A u~^uu~^ = u~^)- 

For details on inverse monoids, see |12j . 

Let E{M) denote the subset of idempotents of M. Then M is inverse if 
and only if 

Vn € M 3f € M : uvu = u 
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and ef = fe for all e, f (z E{M). Furthermore, by the well-known Vagner- 
Preston representation theorem, inverse monoids are represented by partially 
injective transformations as groups are represented by permutations. 

We recall also that there is a multiplicative partial order (the natural 
partial order) defined on M by 

s <t li s = et for some e € E{M). 

Let p be the congruence on A* generated by the relation 

{{uu^'^u^u) I u € A*} U {{;uu'~'^vv~^ ,vv~'^uu~^) | u, t; G A*}, 

known as the Vagner congruence on A*. The quotient Ma_ = A*/p is the 
free inverse monoid on A, and vr : A* — )• Ma denotes the canonical homo- 
morphism. Note that vr is matched, since u~^7r = {utt)~^ for every u & A*. 
Moreover, there exists a canonical homomorphism a' : Ma — Fa such that 
cr = vrcr'. In fact, {up)a' = u for every u ^ A* . 

Two normal forms for Ma were introduced independently by Scheiblich 
|16j and Munn [lOj . following respectively an algebraic and a geometric 
approach. We can make them converge with the help of the Cayley graph 
Ca of Fa- Indeed, let Ca have vertex set Fa and edges g — >ga for all g £ Fa 
and a £ A. Note that Ca is an inverse graph: {p, a, q) is an edge of Ca, so 
is {q, a^^,p). Such edges are said to be the inverse of each other. It is well 
known that, if A 7^ 0, the Cayley graph Ca is an infinite tree (if we view 
pairs of inverse edges as a single undirected edge). 

We can identify Ma with the set M'^ of all ordered pairs (r,^), where T 
is a finite connected inverse subgraph of Ca having both 1 and g as vertices. 
If we view T as a finite birooted tree (for roots 1 and g), we have the Munn 
representation. If we focus on its set of vertices (a finite prefix closed subset 
of Fa), we have the Scheiblich representation. Now Fa acts on the left of 
Ca in the obvious way and the multiplication on can be given through 

{T,g)-{T',g') = {TUgT',gg'), 

and inversion through 

{T,g)-' = ig-'T,g-'). 

Finally, the homomorphism vr translates into vr' : A* — > M'^ given by 
UTT = (MT(n), licj), where MT(n) (the Munn tree of u) is the finite inverse 
subgraph of Ca defined by reading the path 1 — >g in Ca- We shall usually 
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identify ua with u and MT(ii) with the prefix-closed set T{u) of Ra having 
as elements the reduced forms of the vertices of MT{u). 
Note that 

r(MM"^) = T{u) 
holds for every u A*. Moreover, for all u,v & A* , we have 

up < vp ^ T{u) 5 T{v) A u = v. 

Also, if we view MT(u) as an automaton Au with initial vertex 1 and ter- 
minal vertex u, then 

L{Au) = {v £ A* \ v>um Ma}. 

Such basic properties of Ma will be used throughout the paper without 
further reference. 

Finally, we define the norm ||n|| = max{|u| : v S T{u)} for every u € Ma- 
2.3 Chomsky's hierarchy in Ma 

Following [1], we can define rational subsets of Ma with the help of the 
homomorphism vr: we say that X C Ma is rational if X = Ltt for some 
rational L C A*. This idea can be extended to higher classes of languages 
in the Chomsky's hierarchy, for instance we say that X C Ma is context- 
free (respectively context-sensitive) whenever X = Ltt for some context-free 
(respectively context-sensitive) L C A*. 

Actually, in the case of rational subsets, the concept is independent 
from the homomorphism considered, and this property holds for arbitrary 
monoids. 

Anisimov and Seifert's Theorem (see [1]) states that a subgroup of a 
group G is rational if and only if it is finitely generated. We note that there 
is an analogous of this theorem for inverse semigroups [91 Lemma 3.6] stating 
that a closed inverse subsemigroup of an inverse semigroup is rational if and 
only if it is finitely generated (here closed meaning upper set for the natural 
partial order). 

3 Fixed points and periodic points 

Given a monoid M, we denote by End(M) the monoid of endomorphisms 
of M. Given € End(M), 

Fix{(p) = {x € M : xtp = x} 
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is the submonoid of fixed points of and 

Per((/5) = U„>iFix((^") 

is the submonoid of periodic points of (p (note that Fix(</?™') C Fix((/5") 
whenever m\n, hence (Fix((/3'^))(Fix((/9^)) C Fi:K.{ip^^) and so Per((/?) is indeed 
a submonoid). Moreover, if M is an inverse monoid (a group), then both 
Fix{ip) and Per (99) are inverse submonoids (subgroups). 

Let 9 G End(FA). By [5], Fix(6') is finitely generated. Now, by Nielsen's 
Theorem (see [8]), every subgroup of a free group is free. Recall that the 
rank of a free group F is the cardinality of a basis of F. By [7], every 
subgroup Fix(^") has actually rank < \A\. According to Takahasi's Theorem 
|20| . every ascending chain of subgroups of bounded rank of Fa must be 
stationary, and so must be 

Fix(0i') C Fix(e2!) ^ Fix(03') C . . . 

Since Per(6') = U„>i Fix(6'"'), this provides a proof for the following well- 
known result: 

Proposition 3.1. Let 6 G End(FA). Then Per(6') = Fix(6l^) for some 
N>1. 

The minimum integer N such that Per(0) = Fix(0^) is called the curl 
of 9 and is denoted by Curl(0). 

Lemma 3.2. Let ip G End(M^) and let N = Curl(^). Let k > 1. Then 
Cuvl{lp^^) = 1. 

Proof. We have 

Fix(^^'=) C Per((^^'=) C Per((^) = Fix(^^) C Fix(^^^), 

hence Fix(^^'=) = Per(^^'=) and so Cur^^^'^) = 1. □ 

Assume now that (p £ End(M>i). Then ip induces an endomorphism 
'ip G End(F4) defined by ^ = {a')~^ipa' . It is straightforward to check that 
^ is a well defined homomorphism using the fact that (p sends idempotents 
into idempotents. 

Lemma 3.3. Let ip G End(M^) and let u G M^. Then the following 
conditions are equivalent: 

(i) u G Fix((/5); 
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(ii) uu~^ € Fix((/3) and u E Fix((^). 
Moreover, in this case we have {T{u))Tp C T(u). 

Proof, (i) (ii). The first part follows from Fix(99) being an inverse sub- 
monoid of Ma- On the other hand, we have 

uTp = u{a')''^ipa' = wpa' = ua' = u. 

(ii) =^ (i). We have uu~^ > uu~^, hence {uu~^)ip > {uu'^^)ip = uu~^. 
It follows that u(p = eu for some idempotent e > uu~^ and so 

u(p = {uu~^u)(p = uu^^eu = uu~^u = u. 

Finally, let v S T{u). Then vv~^ > nn~^ and uu~^ G Fix(93) yields 
{vv~^)ip > {uu~^)ip = uu~^. Hence T{vLp) = T{{vv~^)ip) C T{uu~^) = 
T{u). Thus 

vlp = v{a')~~^ifa' = v(pa' G T{v(f) CI T{u) 
as required. □ 

Let ip G End(Myi). An idempotent e G E{Ma) is said to be (p-stable (or 
simply stable when the endomorphism ip is clear from the context) when- 
ever the orbit {cip^ | n > 0} is finite. It follows that {eip^ \ n > 0} = 
{e, e(/9, . . . , eip^~^} for some p > 1. We call the smallest such p the period of e 
with respect to ip. By minimality of p, the idempotents e, eip, eip'^ , . . . , eip^~^ 
are all distinct. Finally, we define 

p-i 

IC^{e) = lleip\ 

i=0 

St<^ = {a ^ A \ aa~^ is (/9-stable}. 

We show that St,^ 7^ if there exist nontrivial fixed points, but first we 
prove a simple lemma. Given u,v £ A* and a vertex p of MT(u), we say 
that MT{v) embeds in MT{u) at p if pT{v) C T{u). 

Lemma 3.4. Lei p> G End(-M^) and n G M^. Lei p—^q be a path in 
MT(u). T/ien MT(?;93) embeds in MT{uip) at pip. 

Proof. Considering p in reduced form, there exists a path l^^p-^q^^u 
in MT(ii) and so pvw > u in Ma- Hence {pvw)ip > uip and so T{{pvw)ip) C 
T{wp). It follows that 

{pTp)T{vp) = {pip)T{vip) C T{{pip){vip){wip)) = T{(j)vw)ip) C T{uLp) 

as claimed. □ 
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Proposition 3.5. Let Lp e End(M^) and let u G Fix(Myi). If a £ A labels 
some edge in M.T{u), then a S St^. 

Proof. Let p-^q be an edge of MT(n). By Lemma 13. 4| MT {{aa~^)(p"') 
embeds in MT(n(/3") = MT(n) at p(p" for every n > 0. This bounds the size 
of MT((aa~^)(^"), hence the orbit of aa~^ is finite and so a G St^. □ 

We prove yet another simple lemma: 

Lemma 3.6. Let ip € End(Myi) and let n > 1. Then St^pn = Stip. 

Proof Let a G St<^n. Then (aa~^)(^"'' = (aa"^)<y9"(''+*') for some A; > and 
p > 1. Hence the (/?-orbit of aa~^ is finite and so a G St^. Since the (/9-orbit 
of aa~^ contains the (^"-orbit, the converse implication follows. □ 

The set of <^-tiles is defined by 

Tp = {IC^{aa~^)a | a G St<^}. 
Theorem 3.7. Let ip G End(MA) be such that Curl(^) = 1. Then F[x{{p) = 

V ■ 

Proof. Let a G St^ and t = IC^{aa~^)a. Let p denote the period of aa~^. 
Since aa~^ > }C^{aa~^), we have tt~^ = lC^{aa~^). By Lemma [33] applied 
to we know that M.T{aip^) embeds in MT{{aa~^)ip"') and therefore in 
MT(t). Thus a(p"' labels a path in MT(t) for every n. Since MT(t) is a finite 
tree, it admits only finitely many paths of reduced label, hence the orbit 
{aip"- I n > 0} must be finite. Since ip^ = Tp"', it follows that a G Per(^) and 
so a G Fix(^) since Curl(^) = 1. Hence aa~^ > {aa~^)ip and so 

p—i p p~i 

{tt-^)ip = (J[{aa-^)ip')ip = '[[{aa-^)ip' = (00^^)99^ = tt-\ 

i=0 i=l i=0 

Since t = a £ Fix(^), it follows from Lemma 13.31 that t G Fix((^). Hence 
T^p C Fix((/?) and so 7^ C Fix((^). 

Conversely, let u G Fix((^). Write u = ai . . . Om with aj G A. For 
i = 1, . . . , m, let tj = ICip{aia~^)ai. By Proposition I3.5|, we have Oj G Stt^ 
and so G 7^ for every i. 

We show that u = ti...tm in Ma- Indeed, by LemmalH^ MT((aja~^)(/9") 
embeds in MT(n(^") = MT(n) at (oi . . . aj_i)^" for every n > 0. Hence 
the orbit {(ai . . . aj_i)^" | ?^ > 0} must be finite and since Curl(^) = 1 
we get oi . . . aj_i G Per(^) = Fix(^). Thus (ai . . . aj_i)^" = ai . . . Oj^i 
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and it follows easily that MT(tj) embeds in MT(u) at ti . . . and so 
T{ti...t^) CT{u)._ 

Now, since = ti, it is not difficult to check that ti...tm < u, from which 
the converse inclusion follows as well. Xogether with ti . . . tm. — oi . . . Um — 
u, this implies u = ti . . . in M^. Therefore Fix{{p) = T*. □ 

We can now solve completely the case of periodic points: 

Theorem 3.8. Let ip E End(Myi). Then Per (93) is finitely generated. 

Proof. Let N = Curl(^). We show that 

PerM = (U VO*- (1) 

fc>i 

Indeed, let u G Per (93). Then u = mp^ for some k > 1 and so u = uip^^. 
By Lemma 13.21 we have Cur^^?^'"') = 1, hence u G Fiyi^ip^^) = T*Nk by 
Theorem 13.71 

Conversely, we have T^ivk C Fix{(f^^) C Per{ip) for every k by Theorem 
13.71 Since Per(93) is a submonoid, ([T]) holds. 

Now it suffices to show that B = Ufc>i T,pNk is finite. Indeed, if t € T^ivfe, 
then t = K^Nk {aa~^)a for some a G St^jvfe . Since there are only finitely many 
choices for a € A and the <^-orbit of aa~^ is finite for every such a, it follows 
that B is finite and so Per (99) is finitely generated. □ 

Corollary 3.9. Let ip G End(M^). The following conditions are equivalent: 

(i) Fix(99) is infinite; 

(a) Per (93) is infinite; 

(Hi) Fer {ip) ^ E{Ma). 

Proof, (i) (ii). Trivial. 

(ii) =^ (i). We build an infinite sequence (e„)„ of (distinct) elements 
of Fix(</?) as follows. Let n > 1 and assume that ei, . . . ,6^-1 are already 
defined. Since Fer{p) is infinite, there exists some u G Per(93) such that 
||ii|| > ||ej|| for 2 = 1, . . . , n — 1. We have uu~^ G Per{ip), say {uu~^)p^ = 
uu~^ . Clearly, we can take e„ = lCip{uu~^) = nS)^(^^~"'^)v* € Fix((/3). 
Since He^H > ||u|| > ||ei||, then 7^ Cj for i = 1, . . . , n — 1. Thus we build 
an infinite sequence {en)n and so Fix((^) is infinite. 

(ii) =^ (iii). By Theorem 13. 8| Per (99) is finitely generated, and every 
finitely generated submonoid of E{Ma) is finite. 
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(iii) (ii). If u € Per{ip) \ E{Ma), then u* is an infinite submonoid of 
PeT{ip). □ 

Given ip G End(Myi) with Tp injective, we may write (p = Tp. Recall that 
we need injectivity to extend Tp to the boundary of Fj^. If there are no 
nontrivial fixed points in (p^ a good deal of the hierarchy collapses: 

Theorem 3.10. Let 99 G End(M^) he such that Tp is injective and Fix(ip) = 
1. Then the following conditions are equivalent: 



(^) 


Fix((/9) 


is context-free; 


(n) 


Fix((/9) 


is rational; 


(iii) 


Fix((/9) 


is finitely generated; 


(iv) 


Fix((^) 


is finite; 


(v) 


Per((/?) 


is finite; 


(vi) 


Per((^) 


C E{Ma). 



Proof, (i) (iv). Assume that Fix{{p) = Kir for some K Q A* context- 
free. By the Pumping Lemma for context-free languages, there exists a 
constant p > 1 such that every w (z K of length > p admits a factorization 
w = wiu>2U>3UiiUJ5 such that: 

• \W2W3W4\ < p; 

• W2W4 / 1; 

• wiW2WsU)^W5 G K for every n > 0. 
For every m > 0, let 

Am = { {u, v) G Fix((/9) X Fix((^) \ u ^ v and 3^ G T{u) 

3w G Ma ■ T{v) = T{u) U qT{w), \q\ > in, \\w\\<p}. 

We show that 

only finitely many A^, are nonempty. (2) 

Indeed, suppose that ([2]) fails. This amounts to say that A^ 7^ for every 
m > 0. Fix {um, Vm) G A,„ and let qm^ Wm be as in the definition of A^. Let 

M = max{||a(/5|| : a G ^}. 
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We claim that 

d{q^,qmW)<2P'^-^. (3) 
Indeed, we have T{vm) = T{um) U qmT{wm)- Hence 

and so 

= Umu:;^{qmWmw:;;^q:^)ip. 

Since qmqm > ^^m^im^ we have {qmq^)^ > (tim^t.m^)<^ = ^imW-m^ and so 
T{qmV) C T{um)- Hence (g]) yields 

r(-ym) = T{Um) U (gmV')?'(«^mV') = T'liim) U {qmJp)T{wm^). 

Now it is easy to see that | \wm 1 1 < P yields | |tf m<^| I < pM. Let a € A be such 
that qmCi G T{vm) \ T{um). Then q„T,a = [q,fyjp)z for some reduced word z of 
length < pM and so qmSP = QmO-z"^. Since > Jn- and can cancel at 
most pM letters from the reduced word qmO-, we get r{qm, qmJp) > rn —pM. 
Therefore ([3]) holds. 

Now, since the completion Fa is compact, the sequence {qm)m must 
admit a convergent subsequence {qi^)m- Let a = limm-j-oo ft™,- We claim 
that a € Fix((^). Indeed, by continuity we have aip = \\m.m^oo <li„J^ ■ Let 
e > 0. Since a = linim^-oo 9i„ i there exists some t > 1 such that 

m>t^ d{qi^,a) < e. 

Moreover, we may assume that 2P^^~* < e. Thus, if m > i, and since d is 
an ultrametric, we get 

d{qijf, a) < max{d(gi^^, qi^ ) , d{qi^ , a) } < max{2P*^-'" , e} 
< max{2P^-*,e} < e. 

Hence OL(p = a and so a € Fix(^). Since \qm\ ^ for every m, we have 
a 7^ 1, a contradiction. Therefore ([2]) holds. 

Suppose now that Fix((^) is infinite. Since Fix(^) = 1, it follows from 
Lemma fS.SI that Fix(93) C E{Ma)- Let m > p. Since Fix((/p) is infinite, there 
exists some w ^ K with > m. We may assume that w has minimal 
length among all such words. 

Since \w\ > \\w\ \ > m > p, there exists a factorization w = wiW2W3W4^W5 
such that: 

• \W2W3W4\ < p; 



11 



• W2W4, / 1; 

• wivo'^w^w'^w^ € K for every n > 0. 

Let u = W1W3W5 S K. By minimality of w, we have | |ii| | < m, hence u 7^ uw 
in M^. We claim that 

{u,uw) G Am-p- 

First of all, we note that u = 1 = w yields W2W3W4 = W3. There exists a 
path 

1 >Wi >WiW3 >1 

in MT(n). We have 

r(n) = T{wi) U wiT{w3) U wi'W3T{w5), 

T{uw) = T{u) U T{wi) U t(;iT(zi;2t(;3tf;4) U wiW2W^w/{T {w^) . 
Since IZ^iw^ptJi = IZ^s, we get 

T{uw) = T{u) U wiT{w2W^wa). (5) 

Note that || 1(^3 1^411 < |'U^2^i'3^i'4| < P- Suppose that |l(Jr| < m—p. Then the 
maximal length of a word in wiT{w2W3W4) is < m, and in view of ||m|| < m 
and ([5]) we get \\w\\ < m, a contradiction. Hence > m — p. Together 
with ([5|), this yields {u,uw) € A^-p as claimed. Since m is arbitrary large, 
this contradicts ([2|). Therefore Fix((/9) is not context-free, 
(iv) =^ (iii) ^ (ii) ^ (i). Trivial. 

(iv) (v) (vi). By Corollary EH □ 

Corollary 3.11. Let if € End(Af^) be such that ip\^ is a permutation with- 
out fixed points. Then Fix(99) is not context-free. 

Proof. Clearly, Tp is an automorphism and Fix((^) = 1. Moreover, Per(93) = 
Mj^. By Theorem 13.101 Fix{(p) is not context-free. □ 

The above corollary provides an infinite class of examples, for arbitrary 
curl > 1, where Fix{(p) is not context-free. 
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4 Radicals 



We introduce now the concept of radical, inspired by analogous definitions 
in other contexts. Radicals occupy an intermediate position between the 
submonoids of fixed points and periodic points. 
Given ip € End(M/i) and n > 1, we define 

Rad„(v9) = {n e Fix{ip"-) \ u G Fix(^)}. 
The following result summarizes some of the basic properties of radicals: 
Lemma 4.1. Let ip € End(M^) and m,n > 1. Then: 
(i) Radn(v) is an inverse submonoid of Ma; 
(ii) Fix((^) = Radi((^) < Rad„(93) < Per (93); 
(Hi) if m\n, then Yladmi^) < Rad„(99); 
(iv) Fix((^) = {/C(p(nn^-'^)n | u G Rad„((/9)}. 

Proof, (i) - (iii) Immediate. 

(iv) Let u € Rad„((/9) and v = lCip{uu~^)u. Then {uu~^)ip'^ = uu~^ and 
it follows easily that )C^{uu~^) S Fix{ip). Hence vv~^ = }C^{uu~^) G Fix((/?). 
On the other hand, v = u ^ Fix(^), and Lemma 13.31 yields u G Fix((^). 

Conversely, let u G Fix((^) C Rad„((/9). Then u = K,^p{uu~^)u and we are 
done. □ 

Radicals may behave better than submonoids of fixed points: 

Theorem 4.2. Let ip € End(Myi) and N = Curl(^). Then RadAr((^) is 
context-free. 

Proof. By Lemma 13.21 we have Curl(^^) = 1, hence Fix((/9^) is finitely 
generated by Theorem 13.71 Hence there exists a rational language L A* 
such that Lvr = Fix(99^). 

On the other hand, since H = Fix(^) is finitely generated by [5], its 
pre-image Ha~^ is context-free by [15] (see also [U Ex. 111.2.5]). We claim 
that 

RadAr((/?) = (L n Ha~^)7T. 

Indeed, if n € RadAr(v9), then u = vtt for some v G L. Since va = vna' = 
ua' = uGH,we get v G L n Ha~^ and so u € (L n Ha~^)'K. The opposite 
inclusion is similar. 

Since L n Ha~^ is the intersection of a context-free language with a 
rational language, it is context-free and so Rad7v((/?) is context-free. □ 
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Theorem 4.3. Let ip € End(MA) be such that Fix{ip) = 1. Let n > 1. 
Then the following conditions are equivalent: 

(i) Rad„((/3) is rational; 

(ii) Radn(v3) is finite. 

Proof, (i) (ii). Assume that Rad„((/5) = Lvr for some L <^ A* ratio- 
nal. Then L = L(A) for some finite deterministic trim automaton A = 
{Q,qo,T, E). Since Fix(^) = 1, we have Rad„((^) = 1 and so L = 1. Since 
A is trim, it follows that z = 1 whenever z labels a cycle in A. 

Suppose now that Rad„(v3) is infinite and write m = \Q\. There exists 
some e G Rad„((/3) with ||e|| > m. Take u G L Ci eTT~^. Then u must have 
some prefix v such that \v\ = ||e||, hence there exists some path in A of the 
form 

go — yq — yt G T. 



If we successively remove all nontrivial cycles from the path qo—^Q, we get 
a path qo—^Q of length < m. However, since z = 1 whenever z labels a 
cycle in A, we have v' = v and so m < ||e|| = \v\ = \v'\ < \v'\ < m, a 
contradiction. Therefore Kadn{f) is finite. 

(ii) (i). Trivial. □ 

The next corollary provides an infinite class of examples, for arbitrary 
curl > 1, where RadAr((^) is not rational: 

Corollary 4.4. Let if G End(M^) be such that ip\^ is a permutation with- 
out fixed points. Let N = Curl((^). Then RadAr((^) is context-free but not 
rational. 

Proof. By Theorem 14. 2| Kadj\f{ip) is context-free. Since 'ip\j is also a per- 
mutation without fixed points, we have Per(^) = Fa, hence Fix(^^) = Fa 
and so Ip^ = Ip^. It follows that ip^ = 1m a- Since Fix(^) = 1, we get 
Radjv(99) = E{Ma) and so I{adi\f{ip) is infinite. Since Fix(^) = 1, we may 
apply Theorem 14.31 thus KadN^'p) is not rational. □ 

Finally, we use Lemma I4.1( iv) to show that Fix{ip) is always context- 
sensitive: 

Theorem 4.5. Let pi G End(M^). Then Fix((/?) is context-sensitive. 

Proof. Context-sensitive languages can be characterized as languages ac- 
cepted by linear-bounded Turing machines, i.e. Turing machines T for which 
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there exists some constant K > 1 such that any word u € L{T) can be ac- 
cepted by some computation using only the first K\u\ cells of the tape. 

By Theorem 13.81 we have Fer{ip) = Fix{ip^) for some m > 1. Let 
^1, . . . ,Am-i be disjoint copies of A, inducing bijections ai : A Ai for 
i = 1, . . . ,m — 1. Let 

Bi = {a£A \ aif' / 1} 

for i = l,...,m — 1. We define a matched homomorphism /3j : yl* — ?> BiUi 
by 

_ r aai if a £ Bi 
""^^ " \ 1 if a eA\Bi 

Let = Curl(^). By TheoremH21 RadAr(v3) is context-free. Hence we have 
RadAr((/j) = Ctt for some context-free language C Q A*. We define 

L = {uU~\u^i){u~'^i){uP2){u-^f32) . . . {uPm-l){u~^Prn-l)u | n € C}. 

It is easy to see that L is accepted by a linear-bounded Turing machine 
T- Indeed, since C is context-free, C is itself accepted by a linear-bounded 
Turing machine T'- We can make T consider all the possible factorizations 
of the input potentially leading to some word of the form 

(using multiple tracks to help), then using T' as a subroutine and checking 
each one of the presumed factors using the definitions of the Bi and 
Therefore L is context-sensitive. 

For i = l,...,m — 1, fix a matched endomorphism ipi of A* satisfying 
= vry?* (it suffices to have aipin = anip^ for every a £ ^, so it really 
exists). Let X = AU Biai U . . . U Bm-ictm-i and define a matched homo- 
morphism 7 : X* — > A* as follows. Given a (z A, let 07 = a. Given a G Bi, 
let oaj7 = aipi. Note that aoi^ 7^ 1 since aip^ ^ 1 by definition of Bi. 

Since context-sensitive languages are closed under e-free homomorphisms, 
it follows that L7 is context-sensitive. Note that aai^ir = aipiir = airip'^ 
whenever a € Bi, so aPi^TT = anip^ for every a € A (if a ^ Bi, then 
aPi = 1 = amp^). Thus we get 

LjTT = {uu~^{ul3i){u~^l3i)...{u/3m-i)iu~^(3m-i)u\u£C}jn 
= {ot-1(w(^)(w^V) • • • ivip'^-^){v^^ip"'^'^)v I V E Ctt}. 

Recall that Ctt = RadAr((^). Moreover, we claim that 

vv-\vip){v~^p) . . . (i;(^"-^)(?;-V™"^) = }C^{vv~^). (6) 
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Indeed, if p is the period oivv^^ , then }Cip{vv~^) = vv~^{vv~^)(p . . . {vv~^)(p^~ 
Since vv~^ € Per((/?) = Fix((/?"*), we have p < m and so Q holds. Therefore, 
by Lemma IdT iv) ■ we get 

Fix(99) = {ICip{vv~^)v I V € RadAr((^)} = Ljtt. 

Since L7 is context-sensitive, so is Fix(99). □ 

5 Open Problems 

We give a hst of natural open problems originated by the previous results. 

Problem 5.1. To find examples of (p £ End(Myi) such that: 

(i) Fix((/9) is context-free hut not rational; 

(a) Fix((^) is rational but not finitely generated. 

Problem 5.2. To show that a basis of Per(0) can be computed for every 
9 € End(FA). 

Problem 5.3. To show that Curl(^) can be computed for every 6 G End(FA). 

If Problem 15.21 is solved, then Problem l5.3l is solved as well: we just com- 
pute basis of Per(6'), Per(6'2'), Per(6'3') until we reach Per((9"') = Per(6'("+^)') 
for some n (and we know we eventually will) by Proposition 13.11 

Problem 5.4. To compute St^ for an arbitrary ip € End(MA). 

If we compute the (/^-stable letters, we can compute the yj-tiles in 7^. 
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